Temple junkies will be satisfied for days with the town itself but the highlight is definitely the mountain.
Khao Wang, the famous old royal palace complex on the mountain has a huge, very elegant stupa, some caves with bats, shrines and animal sacrifices, a well-scuplted, vast gold-plated reclining buddha and a museum. Depending on the entrance you take you may be charged a tourist tax 150B, includes entry to the museum on the top. The main entrance is infested by extremely chubby and impertinent monkeys. You can buy bananas for them from numerous small retailers. But beware, give them attention and they'll try to grab anything you might be pointing at them.
Khao Loung Caves are 2 sets of giant caves, located a little bit north of the city. The first cave set has many Buddha statues amidst the stalatites. The second set of caves features a giant 300 year old tree in the middle, and a peaceful setting I think a monk lives here also. Both caves are home for bats. Free.